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ABSTRACT 



• * " This' E^per , provides a composite picture* of 'sotc 
recent studies of resear<Jh^ Heinonst rationv and curbioaluin developrae'nt ^ 

- progrzTms. Tfie priiaary focus . is ^on ttree--&xudies: "fiss.essing 
Vo;::at\^or^l Education Eesearch and Development" (Nat ional 'Acaderay^ of* 
Science?, 1976., COVERD R^portv ED. 128 754); Anthony H. Pa3cal, et 
al. , "tedetai Ptogram*^ ^Vpportijfg -nE ducat ion al Change/ Vol. Ill/ The 
^ Procfess 'of Change, Appgiidix D. Innoyations 'in^ Career Education" , ' • 
.(Rand, 1975, "EDO08 328); apd "An Evaluation etf 'Vocational Exe'raplary 
't^rojects: Part P Vocfa^xo^al Education Act Ajnendtaents of 19&8" ' ^ 
'(DevelopiH^at Associates, Inc, , 1975, ED^ 109 475). ./Two otlier studies 
ate used for coViparispnj; "Report, to the Congress: .Hhat is* the Role of 

-Federal ' Assistance for vocational Education?" (Comptroller General of ^ 

• ihe yftited States, 1974,^ GAO Repc^rt,»Ep 105 132) and "Knowledge dnd^ > 
Policy in Manpower: Study of the . fj^npower. Research and 'Developinent , 

^Progra^y in the'^Depij^rtipen^ of Labor" (National .Academy^ of- Sciences, ^ 
.1975, ^ DOL/,MRSB Report, ED 118 80!^). Each re'pd'rt is reviewed,'' 
jncludx4ig methods of > 'research, findings, conclusions and - 
recommenda^iions-; and a critique; All ^ve reports are . summarised, in 
tabie' €omp aring and. highlighting ' purpose, scope, data sources, . - ' 
findings, 'and .recojamendatibns.*-A synthesis- of the repori:s is »also 
presented* Recoiamend^ti-ons /f o^ me^thods of evaluating futur^ , 
Vo^tionai- Education R&D pi^ograms ccnclude the document. 'A^ sho;|>t* ' 
bibliography is appended^ (EL) * * ° ^ * .» ' 
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VocatiOQai education researcl^and devefopment (R&D) enters a new era with the newest federal 
■ legislation for vocational education {P/.L 94 4S2). Xhis legislation has been m-fluapced by the strength 
and weaknesses of the vocational education R^D program conducted since passage of the Vocational 
tducatioo Act of 1963 These strengths and "weaknesses have been g^cammed by the Comm.ttee 
on Vocational Education Research and D^elopment (COVERDrin a USOE 'sponsored study managed 
t^v he National Academy of Science. This and four^ther studies are.tKe subject ef this mtegrative ~ 
analysis. : - ' , ' ^ a ^ 

Evaluation ofyocational Education R&D ProgYams: An integrafive Analysis of Recent Studies 
was prepared, at USOE request, as part of the AIM^RM Projedt at The Cer^ter^ for Vocational Educa- 
tion It IS part of The Center's continuing effort to'analyze information on issues and problems ' ' 
critical tQ the future of vocati.onal education. TKis ptiBlication treats a subject which is of critical 
importance to The Center-vocational education R&D. Ifreiteratep the weaknesses in the R&D 
program rfevealed in-the assessments and is critical of the assesssnents themselves. Most important it 

^uggests how a future assessment might be designed to have Sven more impact on thefl&D program ' 

^ IS recommended that planning for th^ext assessment begin immediately 

^ Recognition is given to Joel H. Magisos and Allen B. Moore fortheir scholarship as authors of • 
this publication. Or Magisos is Associate Director of Information and Field Services and Dr Moore 
is a Specialist at The Center. • , ■» .V '<..■• 
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INTRODUCTION ^ * 



The purpose of this paper Is to' review and synthesi,ze»ome recent studies a'f ces^earcb demon- ' 
stratton, ahel 6yrricul.una(idevefQi)ment programs funcjed under Part 4(e) of P. L. 88-210 acid Parts 
C, D and i of P.L. 90 579 lt;i<Oi/ides a composijte picture oi these studies and suggests som'fedirec- ' 
tip*is for further assessment of vocatio.ial educatiotoearch and development. The studies.uhder 
• review were" conducted in the context of growing pu% interest in'the effecttveness of governmerit- 
sponspr«j programs, concern by those involved in the B^eration.of these programs, and preparations 
fpr nev* federal legislation for vocational education. Tjfree of thes^studies ar.e the'focus of this paper: 

Ass^-ing Vocational Education- Res^rch and Development.' y^ashmgton DO;- National 
Academy of Sciences, 1976. {COVERD Report} .ED 1-28 6^4 

- Pascal, /Vithony H^etral, Federal Progr^^ms Sopporting Educational Change, Vol. Ill The Process 

' . cn, no^^' ^PP^"^"" ^ lnno\^gdns in Caf-^er Education, Sant^a Monica: Rand, April ^3975. 
•'EDI 08 328 ' * ^ 4-' . . ' > 

An Evaluation of Vocational Exemplary. Projects. "Part D Vocational Education Act Amendments 
of /S^S. Washington, D.C.: Development Associates, Ino., March 1975. ED 109 475 

Two other studies apeconsiaer'^d for context o? cbnlparison although they do not themselves focus on 
vocat-ionaf education research, curriculum development, or, demqnstration programs. Jhese'are: ' 

^ Rejjort to the Congress; \vhat is .the Role of Federal Assistance for Vocational Education? ' ■ 
Washington;,D.C.: Comptroller General ofrthe United States, December 31 1974 iGAG 
Report) ED 105 132~ . ' * , ' . ' ' 



^, . Knowledge and^licy in Manpower: A Study of the Manpower Research and Development ' 
- / '- ^^^ -Oepartmjnl^l Ubpr, Washington, D.C. National A6adem.y of Sciences, 

■ 1975. (DOL/MH&D Reportf ED 118 802 - '■^ - 



Either the acronyms {C0VERD\ dOL/MR&D-, and GAO'lor the publishers (R^nd and Development 
Associates) are used throughout thf paper when referring to these reports or the work of the group 
doing the assessment Reference tc<R&D in this paper is to prograrrjs of^eaearch, curriculum develop- 
ment or demonstration conducted under the^rovisions of federal vocational education tegisJation or 
to research arid development generically. _ • . _ 

~' < t • 

The relationships between the five sVudies is shown in Figure 1. Th&GAO Study provides over- 
^sight of the entire vocational education program. Vocational educatjWs'the link between the GAG '* 
Report and the CGVERD, Development Associated, and Rand rep.orts which consider various aspects 



'Committee on y9'cational Educatjon Researched Development 

-Department of Labor Manpower' Research and Development 

•'General Accounting Office ^ , ' ' 

' ' - . ' 1 
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Vocatiorlal Etiucat^ion 



AO Report 



Pai^B 




•V 



'1 



Vocational Education 
Research and Development 



COVERD Report 



'\Part^ 



Part D 



Parti 



/ 



NOTE: Reference to "Parts" ^ 
Is to sections of - 
■ *.-P,i.. 90.57p. 



Development Associate^ Report- 



* Part D 



r • 
I 
I 
I 



Rand Report , 



PartD . 



Manppwer " ^ ^ 
^Research and. Development ' 



DOL/MR & D Report 


•MDTA 


TA ' 


Other ' 



.r7 



Figure 1. Relationstiip of Reports 
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. ^ ' • A. • ' ' ' " " ' ■ ■ ■ ' . - " " 

/ , ■ . ■ • . • . - • 

of the vocational educa;.on R&D progiam, R&D isnhetommon pDrpose between t>6se reports a'nd 
the DOL'^MR§(D Report which focuses orfthe manpower R&D prog'ra'm of the Department pf Labor. ' , 
. . ^Whi]e th§ Development Associates and Rand,reports*focus on PaevO of P.L. 9b-576, only COVERD 
focuses.on alT aspects Of th4fed.erally-assiste(J»vOcat"iorTal education Ff&D program ' " - . • 

I his paper summarizes the-^ubstance of Ihfese reports and seeksr to find^same common threads' 
• among t^em that might synthesi/^'into conclusions wiJ|;imipUcatiQns for fu'tureprggranis of self 
rene<Val for vocational education. The review for.thisiDaberhas iricludefl^tudy of fhe reports;-4is- 
^ • . cussion with.the investigators and sponsors, and review of earlier drafts bV leaders in vocational edu-' " ' 
cation-and vocational edacation R&D 

■ , , BACKGROUND. Of. VOCATIONAL EDUCATION R&Q ' 

The foundation for vocatmnal education R&D vvas laid with the "Snlitlj Hughes Act of 19t7. ■ ' . - 
AlthpUgh' R&D was i3otiuthdrized>e/-se in-rhis act, it did provitte continuing support.for educatfonal" ' ' 
programs in agriculture, horne economic^, trades and mdustry^-fend teache/.tramrfig. Teacher traintng ' - 
'or teacher educatian asjt IS now calledMoragciculture and homB-econorTi"fcs was conducted, for the " ... 
'.most -part, in land grant colleges where a/ich tradition of ft&D. was developing m these SL^Istantive ^ 
. areas^The teacTier educators,, themselves "a pro'duct of jhe land grant colleges, could see the a'ppli'ca- ■ ' • 
bihty of scientific methods to tHe problems of the developing program! 'Furthermore, they were 
^ employed in settfngs\vhereschol,arly vyork was considered .^ssenti^l to the professional" role of ' 

^. faculty members Many of these teacher educators condij^ted small stl)di,es and directed the-r|search- 
wcffk .of gradugite students. Thfese studies' were virtually "labors of lov«" beeause,they had h,ttle . 
• financial support, hence, they were limited in scope and rarely d^alt with problems of real significance. 

r Other federal legislation was enacted between 1917 and 1946 which extended vocational educa - 
t1on1nto new afeeas (e g .distributive edi>cation),Jncreased the authorization* for funding or dealt With*, 
, ernergencieMe g., World War llj, The George Barren Act of 1-946 was the-first to Spefciticaily mention 
^ research when it authorized fiinding for guidance, teacher training and researdh. Foljowing this legisW 
* ^ ; tfon, vocationar education R&D still was limitetj to>6nnall staff studies and graduate research conducted 
4 in teacher edu^:ation departments of colleges and yniversjties. Few leaders m vocational oducation en^ . * 
^ visloned a.|3r6grarn of R&D*that woyltl tackle v^ationaLeducation problems. ph a national scde. 

. Thfe VocationaJ Education Act of 1963-(P.L. 88'210} prpved to be the landmark' legfslation fo'r , 
>K vocati(*hal education' R&D becDJse it contained broad grqvjsions for research, trainJns^ experimental,- v. 
derndnstration or pilot programs needed lo caVry ou/t'the emphases of the Act. Section 4(clpf P:L.. 
88-210 authorize(tfundin§ these program^ ^ the discretion of the Commis^ioperof.E , . ^ X 

. 10 percent^f th$ tot^^l fi>ids appropriated under the Act, P.L.'88-2l6 also had a di.fferept educational 
program focus It called ^r educational programs designed to meet the needs of special^popufations^df 
^ .< ' P^oPj? rattier thpn focu? upon o.ccup^.tiqaa4 areas. tWuch needed' to be studied and developed^ The fitst 
^Ijig-fGp m vocational education R&D was building the capacity for R&D.. In a series of program 
.administration events/several new institutions for F^'d w^re,^stablfshed with federal funds. V 

♦ The Center for Research and Leadership De,velopmentjri_ Vomi^^^^ 
* • ^ (now The Center'for Vocafional Education) at The Qhio* State University (1965) • • 

• ^RIC ^Educational-Resources lftf9rmation Center) Cleannghotise on Vocational and Technical . 
'Education at Tbe Center at'The Ohio St^te University (1966) \ ' ^- , ^ ^ 
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' % • • • 

\ "Pfte Center forOocypationarEducStion-at Nurth Carolina 5tate C/niversity (19&§) 

■ I R-esearch anfd Development Units (ftDU) at Washington State University, towa'State ?ni>l8r^itv. 
and,l^id\\ganStat^gnivefsity ;i965) • ' ' ' • > ^uf-V 

♦ ' • Research Coordmd^ing Urtit's (RCU) at umyersrtiesprJF state depS^ - 

" most of the states (1 9651 * ' ^ - ^ ^ • S 

* * * — ' *. » ^ ' * • ^ 

Funds from P L,. 88-210 also were alloeat^ fpr.such research pnoriti^s^as prografr^eOaluation voca- > 
tior>al education resourcQ|-ctevelppmen^, yotatitjoal guidance and career phoice,.orqanization aoid 
•administration, and 'hew_carders (Taylor and Miller). - • 

• -.1 nese wer^jexatjngand challenging timeslor'th'ose Engaged m vocational education R&D. Un- 
;f^nately: ;he tide jurned for VDcatlphari^ducamn'R&D wheiji Con^r^ss declined to continue ^the 
.ftmallocafiQn of authori^ kjHds'^for.the R&D pTogfar^). Federal administrators^were foTced to • 
.qlose'out qr eii:astTratl^7ecfcce the pr^pgrams of some of the institutions tHa^t theVhad establishecl. 
The ROUls were-ab^ndoned altogether.. RCU's were cifftailed^tintil r.eaVthonzed yonder the 1968 
Amenflmenjs -The E13IC Clearinghoi^ on Vocational ^nd Tefilinical Educatron was continued under , 
;the general ERIGprogrann. The C^ntSrs became mpw clojely'alig^ed vi^th the univgrsity-based ceriWrs 
ahd pegional educational laboratories in generaNducatic^n. Afrof these mariuevers,vvere acco'mpanied ' 
by an evolving pipgrarn purchase policy in the U.S. Office.of- Ed>icati6n, as Qpposed to an institution 
jcapacity,buildmg^'posture, ^ * ' ' * * - . ' ^ 



The-Vocational.Edi/catifjn Amendments of I^S^P.L. 90 576) gave'vocatiotiaLeducation R&D . 
another chattce, but^in another fcTrni. It auth&rized.tTTesJpport of .RSD under tfiree parts:* ' % 

, Part Q- , res^rch and 'training in yocaHona^^c|jjc^tten, .50% to belacfmiiiistered by the States 
^- Jetwt)ieh 75% couJd-^^e us6d for R.CU's*.) and SQ?I to be ad/Viinistered by tht» 
- ' .'n -^^Commissione^of Education 'y ' ' ' , ^ ^ . 

Part D:»^ "Sxemplary programs and|projects (demonstrat4on)^50% to be administered by the. 
* ' \, /-.'^ States and 50% by the'CommissionQr . ^ » • 

; Part I/, curTiculuim development in vocational ^n^i t*echnical education, 100% to be admini^- 
' ^ ^ ^ tfered'by the C-ommissioner . ^ • * " ^ , 

Insofar as the establishei^ R&D Institutions wer^concer feci, with the exception of the RCU's, 1-he 
196&?kmendments made the prograYn purchaj^.pj?^^ institutions were torced.^- 

comp'ete^for the privilege of^condUc^mg each project unfljer a grai^f or, m'ore corTTmoniy, a'corkr 



to 

fact. 



^ u ^^^^ n^v^est fdd.eraHeg!slatioil for vocational edujcation (P.L. 94-482), pass^ed October 12,' 1976 
««uring the development of l^i^ (Japerl^maintairls/hejntent of.ftartaC, 

mpnts„but it r^uir^^^ojn^ cf^nges in'administf^tion. For example/more discretion'is given tlo the 
V Commissione^r, funds are no\gai|Ti^rked f6r specific kin^^ of R&D activities, and applicants ar^ 



^ , . TheVocational Atfiendments of 1968 (P.L. 90 576) provided func^s to state agencies resu-lting 
in many RCU.Tbeing mov^ from y/iiversiti€s "to s^ate defjartments of education ^ 



required ton". ; , d£monstrdle.d redsonable probabfl^Jthat ^uch gr^t will result \\) improved teaching • 
techniques bf curriculum maJeriaJs ... .'in a substantia number ot classrooms or learning, situations 
within^flve years .^fter the termmatiqn date.af such grant." [P.U 84-482, Set, ib\] It is interest- . 
ing to not6 i\)at Corigress explicifly authorizes the establishment Of a National Center for R^earch * - * . 
in Vocational Education with six bro^d functions: * ' ' ^ / 

* ' ' ' ' \ • ' ' • , 

• ^ "(A) conduct appliec;) reseaVch and development ofi problems oAnational signifi^caiice • >. . - 
* in vocational educatipn; ' y ' - ' •* , / - > *^ • 

< j'(B) provide.leadership development through an advanced 5tudy cefiter and 'inservrce. . ^ - 

education activities i^Jr State,.and 'local leaders in voQ'^flonal education; , * ^ * ' ^ 

"(C) disseminate tfie results of the research and development projects* funded *by the » 
centei-; ^ . • . ^ \\ \' . / , ^ . • ; . ^ 

- v(t)) devel6p and provide intorn^^tion to facilitate national planning and pplTcy "develop- . - /* * ^ 
m'ent in vocational education; ^' . ; " / • : ^ ' . . 

"(E) adt as a. clearinghouse for infqrm^tioh on 'C9ntracfs made'by the States pursuant . ^. " ' 
to section 131, section J3^, and section 133 on contracts made by the ComrT\issioner . - 

pursuant to this^ section; and (ii) compile an annotated bibliography of research, ^ ^ ^ 

.7 exemptary and innovative prograrrj prd^ects, and curricOlum development proj|^ • ^ . • 

a^sis^ed with fuhds made available under this 'Act sWe July \, 1970; and ^ * 

, /''(F) jy.ofk with States, local educational agencies, and other public^gencies in develop- 
' ing methods of evatuating programs, uding.the foltow'^up studies of prograrp com," 

pJeteVs and leavers required^by section 1 12, 50 that these agencies can pfier job training ^ 
programs wraich are mdreclosely related to the types of jobs available in thei^orpmuni- * ♦ 
ties, regions/afid States. [Secticwn I/I (a) (2)] . ^ ^ v ' ' ^ . ^ 

. 4 ' REVIEW OF REPORTS , • * . 

. - - - ^ , , ••'-4 ' • • . , 

'< . . / • . 

The reports'undeh<ewcw h^ an impact on new legi^ation. Too, the same Go'nditions^and opin- 
Jons influenced both the investigators (i.e., COVERD, Rand„ Development Asso^ates) ^nd Congress. 
The five reports described in the foljdwing sections are important to those developing vocational 
education RAD programs under the new Act. At least one of the reports has afreacly generated con^ * 
siderable debate among those who conduct vocational education R&D. Controversy is to be expected 
when an evaluation is conducted on such a broad scale. R^g^d^rs^should try toy^few ttie reports objec 
tlvelyand keep in mind that even members of COVER D only . ^agreed to dfisagree . . ."on some 
points (COVER D, p. vi). /\gree ocydisagree, the reader should look for implications in what has beeo' 
learned about vocational educatioji R&D and what others think about it. • « . 

M h)e three pr]mary and two related repprts are summarized h TaWe I, highlighting purpose, scope, 
data sources, findings,, anjd recommendations. Detailed information is.avaiiable in the reports th'enn* 
selves, each fully cited in the BibliogVaphy with ERIC document numbers. The full text of the reports* ^ 
IS available in microfiche or hardcopy from the ERIC Document Reproduction Service. 

- ' ^ C'OVERD'Report' ' ' , 

The Committee on Vocational Education Re^ea/ch and Development (COVER D) was formed 
by the Natipnal Academy ofSciences to review and assess the vocational education R&D programs 



^ Assesswg Vpcatioi^f Education Re^arch an(i Developm Washingtort, D.C.. Natronal ' ^ 
Academy of Sqf5nw<T976. . » ^ 



> -TABLE I 

' • ^ COMPAR.ISON Cy^ VOCATIONA,l>EDUCATJON R&D EVALUATION 
# ' AND. RE LAltD REPORTS ' 



Comparison Factors - 


* , * ' 'Vowtibnal Ediication Evalualion Reports 


Related Reports 


/ • COVEFfD/ 


' .RAND. 


'iDevelopmeht Assogiates 
* ,^ * 




. D.OL/l\^&D 


1 . Purpo^f of , 
Study 


' Review and assess v^oca- . 
tionat education R8iD^ 
recommend dhangjes in 
poficy 


Examine federal pro^r^s* 
^ supporting educatlenal 
changeMncltiding voca- < " 
tiopal edt/cation ^ 


Obtain'inforijiat*on to assist . 
continued operation of vo 
national exemplary pcoject§ 


* Review ^>peratiQn of vo- 
'^cational eflucation pro- 
grams 


Review, assess, and rtiake 
recommendations r^ardmg, 
ma/ipower research and cle- 
velopment' ' * 


2: Scope of ^e 
- * Study Topic 

:\- .. 


Pro-ams funded unSer 
Part 4 (c) of PL 88 210 
and Part'C, D, and 1 of 
PL 90-576 dunng'l965- 
1974 period, S250*nnil- 
honjfor nearly 5,000 
projects 


• 

First round of career edu • 
cation exemplary projects 
' funded under Part D o J , 
PL 90 576; $2?2y2 million ^ 
(Si 20 thousand per year 
for 50 pfojects) i , 


F^rSt round of career educa 
tion exemplary/ prpjects 
funded under Part D of Pt 
90-576, $22!4 million 
.($108-200 thousand e^ch 
year/state for about 61 
projects) * ' 


^ 

Vocational education 
programs funded since 

• enactment of PL 88-210; 
$3 biMion toi;af^($146 ^ 
tnil]ion in.the seven * 

stales studied in 1*973) 

• ^ 


R&D programs conducted 
by the Department of 
Labor's OMRD undeL, 
MDTA.an'd CETA^S^O 
million f^ nearly '2,000 

pJrojects ' 

1 * 


3. Invesj^ttng , 
* organl^atl^M^, 
sponsoring^^ 
agency; ddfi^ , 
pf report <^ 


National Afcatjemy/Of 
Science, U.S.OTftce of 
E8u9ation/BOAE, I 
August 1976 


Rand, U.S.*Officeof 
Education, April 1.975 

• 

: <^ ^ 


Development Associates,* . 
Inc., U.S, Office ol Educa * 
tion/Off^ce of^Planrfing,* 
Budgeting, and Evaluation, ( 
March' 1975 " 


} 4 

Comptroller General 
of the United States, 
Qongress, December 
31,il974 

: — . — — 


— t ■ — 

National Academy of 
Science^ U.S. Department 
of Labor/OMRD^ 1975 


4. 'Sources of 

Data and 
./ Methsodolggy * 

. .■ 


Committee merrtbers, 
testimpr^y apd survey 

of •fp;^rJpr^ And rpnrp^pn- 

tatives, 15 commissioned 
papefrs; s'rie visits to 10 
RCU's; review of re- 
ports and literature ^ 


Revtevvof literature. 
Site visit to.mne projects 

^^rounri tilP rnuntrv fniir? 

state-administered,«4ive 
federal-administered 

> 


Ou^sfioitnaires and tests to 
4,632 piirticipating and 

4 flfi*? nr>iin;irtirin;»tlnn fith 

9th, and *J2th grade stu^nts 

Jn 50 projects, a?rd to 1,4^3 

teachers and 229 counselor 

participants plus a random 

sample of nonfiarticipating y 

6th grade teachers 
< 


Program reviews in 
California,^ Kentucky, 

tVIII II ICdU Ld, wiltU, 

Pennsylvania, Texas, ' 
and Washington 


Review of literature, com- ^ 
mittee input, 20 ^mrnif 

Visits ' • f 


5. Findings ai^d 
conclusions of 
study 


: — ^ — ^ — 

Little ^v»dence of im 
paat on students^due ' 
to"- 

Shifting priorities 
Geographic distri 
bution 0"f Ju^ids 
Lack of coordination ' 


' Projects vveak as a treat- 
ment to ^votd r^sis- 
tance;i)ecame short- 
-lived^, insignificant 
^ Enrichment programs ^ 
Projects planned bettefr^ 
did better 
'if 


Little relationship between ' 
objectives and performance 
activities ' 
Little impact on students 
Lack of drfference may. be 
'attributed to use of treat- 
ment In Qonparticipating " 
, Jbhools 


State and local support ' 
mcreas^^ ^nrallment 
grown; opportunities^ . 
for disadvantaged ^nd 
handiccg^ped Increased 

JLJse of federal furrds not * 
adequately evaluated 


Accomplishments in four 

areas - ^ 

Labor force data/ ' 
l^febr fnarket theory 
Manpov\/er needs of dis 
advantaged ^- 

; ' ' * . 1 - 
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TABL'E I (Continued) 



Comparison ih^ctors 



Findings and 
conclusions 
of study 
(cont?f»ued) 



Vocational Education Evaluitjon Reports 



COVERD 



Inadsqua'te dfssemma 
tior^and utilization 

Failure to maxinr)fz^ or 
measure impact 

Slow start up 

Failure to touch real 

* questions or clarify 
issues 

More development and 
demonstration than 
research - ■ ' _ 
Body of knowledge m- 
er eased, programs and 
products m use, and 
R&D capacity built 



RAND 



Site selection essential 
to success; middle- ' 
sized schools best 

Project directors'must 
Jt)e good performers; 
school personn^^eed 

•tg be involved m plan- 
rimg ^ 

Materials developed on 
site work better 
Extensive staff develop- 
ment neededc 
LiHle SEA input 
Little C^'oss fertilization 
Staff dissipated m third 
,ye'ar; little persistence 
.of pregrams bev^ond - 
project . 

Evaluation had little 
tiTipact ^ 



Development Associates. 



Stt/dent outcomes related 
to expenditures; many 
projects under-spent 

Not all treatments used on 
all students in participat- 
ing jghools 

Required contirujation of 
activities beyond, projects 
did not occur 



Funds used to support tradi- 
tional programs too often 

Federal iliii! ii %<)iiri ii nof 
useS effectivefy* ' 

Planning often has bebn too 

•corhpliantrhmited and data 
starved 

Trainiggtesources'not con- 
• stdered • 

Facilities not used efffcierrtly 
ManpowerrequiremeQ^ pot 
considered ^ 



■ Relate^ R^pprts ^ 



* ,GAO 



t)OL/MR& D 



^ Manpower policy and 
program effectiveness 
.(assessment),. 
Under emphasis in six areas 
Unempldyment and labor 
* shortage^ * 
Cultural factors 
Job search behavior 
RacteJ discrimination 
No«treatment var'bble * 
Worker dislocation and 
« adjustment' < 
CohereiyR&D strategy, 

^hougfCl^cking a central, ^ 
, unifying thrust 
Quality of intramural R&D 
, vafied,,but generally good 
Declming size and experience 
' 9f OMRD staff adversely af 
fecting R&D maijagement * 



6 Recornrnen- 
dotion in 
report 



ERLC 



Consolidate Part C. D.»and 

1 programs 
Administer R^TD in BOAE 
50% for national and multi 

state prionfies 
Separate and articulated ' 

career education R&D 
Improve pnonty setting 
Mix announcement and 

funding procedures 
Involve women and mmor^ 

ities in B&D 
Adequately fund a national 

R&DcenteY 
CoQJuiihate cufTiculum de- 
velopment, possibly at 
' National Center* 
Adequately support ERIC 
^^nd AIM/AiRM in oqe lo- 
• cation 

Establish information 
analysis program 
' Fund dissemination and 
iftilization-; using several 
strategies 

U& ^ > 



(No recommendations, 
conclusions for use in 
improving subsequent 
rounds of protects are 
implicit in findings) 



Implicit 10 the findings 
and conclusions ar$ recom- 
mendations that predeter 
\ iTi";>^(tt^nteri^ be gstab-^ 

\lisheer 

Gyidance-yid^sTsistance m 
project managfemenj be 
given 

dear intent, careful plan 

ning\ and sufficient - 

staffihg'.b^ provi^Jed 

Comprehensive^ impact- . 

oriented evalQation be 

used for program modi- 

ficatioOvV ' * 

• > \ ' t 



Set limit on use of f^deraT , 
funds for state administra- 
tion* 

Require federal funds to be 
useci for planning at state 

Requffe thatjederarkunds be 
Osed to develop, improve, 
and/extend ^ ^ 

Addpt policy' to provide 
special needs programs by 
requiring either specific 
matching or larger set- , * 
asides » 

Require relating to post- 
secondary commissions 

.andCETA * 

Require set-aside for coo^- 1 
erative use o^ facilities 

Avotd construction * * • ' 

Match programs to job 
market 

Require work experience 

Require placement and 
follov/-up ♦ 



^Co-exisjSnce and interaction 
of program components 
aimed at different policy 
laterestsand R&D*objec-' 
tives 

Areas of conceptrated effort 
related to long-term policy 

^Flexibility to respond tO' 
knowledge requirements at 
policy and prpgram levels ^ 

Balance arruDng, sNort, 
medium and Igng term v * 

' R&D V . 



1 



sponsored by the;U.S. Office of Education during 1965 1p74 under authority of the Vocational 
Education Act of 1963 and 19.68 Amendments, and to recommend'changes in R&D policies and 
programs for the coming ;iecade. The 1 1 member^ of COVERD were selected because 6f.their 

Jew! cbmpetence and representation of R&D, vocation^ education and related fields. Their work -< 
> sopported by Academy staff, consultants, and USOE program administrators. Their "report was 
. ..lewed by a group ot^ier than the authors and was,approved by thl? GoverningTBoard of the National 
Research Council, whose members are drawn from the Councils pf the National Acadenty of Sciences 
-the National Academy Of engineering, and the Institute of-Medicine. Initially, the report was to have' 
been avertable for Congressional hea rings jon. the neW vocatfooal education 'legislation. As it turned " 
ouj^only a draffof the report wasavaifaljle foi* Congression^ committee hearings. 

Methods ^ • ■ 

• > 

^ COVERD met many times to grapple with thf^ssi>es and tp analyse a mountain of cfeta Ttieir ' 
' methods included : . ^ • ^ ' ^ 

* 1 ' Reviewing 120'reports 6f vocational education R&D and other^Uter^ture such as reviev^ 
3nd syntheses of research. . ' ^ 

2. , Commissioning 1 5 special reports and papers related to vocational educaXip^&£), 

3. ^ l;jfolding hearings to actept^oraCand j^ritten tesjimony from venous vocatfoaal education and 

ancj vocational education R&D leaders and surveying reprfesentativ^of'groups {some 20 
represented)j , * . ^ * r 

4. Providing their own input t9 the Cdmn^iitte^eliberatipns. ' • ^ 



i 

5. 



Visiting. RCl^'s to observe th^ state administration of vocational education R&D lil^ds 
particularly Parts C and D. * ' 



4. 



6. ^ ^yntKesizing'a?! of the input into the C@VE^D Report. 
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Findings * - 

The first paragraph jyf the report *is startling in it& directness and imp|jcat;ions: 

The Commit^ hasTound that tFte'&250 milliQn spent {jy the U.S. Office of Education 
on vocational education research and development during^the last ten years has not had 
documented, widespread impact. Althgugh'the committee did not have adequate data 
♦ and models for rigorous evaluation, the available data do not indicate that vocational 

education research and development (R&D^ findings and products have had aa influence 
^ on the knowledge, skills, or employability of large numbers of students. Tb^ommittee 
believes thaf vocational education R -& D shares with educational R & D a lack of both 
demonstrated i?npact on stildents and methods for measuring impact (COVJftD p 1) 

COVERD highhghtkl s^eral of the reasons fcr this limited impact, ihcludmg shiftmg,fesearch pnori- 
ties b^sed on p^olitical and bureaucratic considerations, geogr^jhic restrictions on distribtrtio'n of R&D 
funds, l^ck of coordination between parts, inadequate dissemination and ut(hzatipn, failure to maxi- 
mize or measuce imp.act,'and slow^tartup because of .the need to build capacity COVERD believed 



that the deficienct^ stem from a lack of {johcy, administjation, and leadership in vocational educa- 
tion R&D. It noted that, in spite of vocational education's shift in focus' fyom la'bor market to 
peopje, R&D rarel,y touched on the real questions or helped to clarify the issues. CO-VE^RD was * - 
hampered in its ov^^n'^ssessment by a lack of objectives in R&Dr insufficient data, and subjective'- • 
impact measures. COVERD recommends some sweeping changes to effect communication and 
coordination, ensure work on'iong term problems^and heeds Qf special popujptions, minimize pohtt- 
tal and bureaucratic influences, improve information handling, and increase dissemination and utiliza- 
tion. • , * * . * 

COVER [5* reviewed the history of education and vocation^ education R&6 and concluded that:- 

- ■ " - ^ ^ / ' • ./■ 

if research is to improve the educatio/i of vocational studen^ts it must be more farsighted, 
expanded in scope, and improved'fn qgality^( COVERD, p. 15)* 

COVERD furt^r concluded that thdfe baj been rYiore development and dentonstration than research, 
research has been more descriptive th^n e^^perimental, there-are few researchers, career education ^s 
received .emphas^s^'at the expense of vocation^f sdrfcat^on, and there has been much activity m cur- 
riculum development. It reviewed t>ie findmgs and methodofegical problems of Project Baseline* and 
Development Associates (also reviewedin this pa^er). It described nioe projects considered success 
ful by USOE personnel, the Southwide Research Coordinating Council, and'indwicli/ij COVERD mem 
bers: ' \ ' ^ 

Aviation Mechanics Project (Allen) . ^ * 

Electro-Mechanical Equipment Technology Project (Roney) 
. • The fCingdom of Could Be You (Sutherlant;! Associates) S ^ 

Alabama Vocational Management Information System \ . 

(Florida's Ecpldgical Approach . » • ' T 

' Mississippi'-Career Education Project - .y* 

Texas Survey of Needs . ^ ^ 

Allied Health (Anderson) * . ^ ' 

Job Experience Kits (Krumboltz) 

'Implicit in the COVERD report is that, of 250 millfon dollars spent in 10 years, only part of the 
funds wete for'research because of' the catego|f!cal allocations to research (Part C)', demonstration 
(Part D) and curriculum development (Part I). Further, even Part C research funds were li^&d for 
other than research (e.g., evaluation, denionstrajtior/, etc.). ' \ - ^ 

COVER D.^s^mal chapter is devoted to a deserfption of national and state levels of organization 
, Invok/ed in the administration of the vocational education R&D program. On the national level, th6 
administrative location, coordination, planning and priority setting, policy development, grants and 
contract award, award recipients, project monitoring and evaluating, and dissemiijation by the U.S. 
OffiQe gf^Education are described. Aisaon this level, the national R&D centers. National AdvisQry J 
Council for Vocational Education, and national dissemination systenrij (ERIC and AIM/ARM) ar^ 
described. Attention is givep to the regionally based National Network for Curriculum Coordination 
(NNCCVTE). The*RCU's, state departments of vocational education, end state advisory councils 
are discussed. Dissemination and utilization is discussed separately. 



*Lee, A. M., Learning a Living Across the Nation. Project Baseline.^ Flagstaff, AZ."" Northern 
Arizona University. 



Conclusions and Recommendations * ^ / * > 

WljHe' lamenting the lack of evjdence upon yvhich to base its assessment, COVERD concludes ' 
,tbat vocational education R&D has added to the booy of knowledge, has produbed programs and • 
techniques now in use across the nation, and jjas built its own capacity during this relative short 
period. It recommends that vocational education R&D objectives be clearly defined, reports arid 
research syntheses be more accessible, and a plai^ for evalua^i6n be developed. 1^ recomrhends an 
advisory panel be convened every^five years for review and assessment of the R&D programs. 
COVERD developed concliTsions and recommendations wtiich are^paraphrased in the {ollowing . 
sections. • ' . , 

Program Structure ' * - ^ ^ , 

»• * - • * * 

• Rdsearch conducted under Part C is not being fully us^d as a basis for curriculum developed 

ucjder Part I and for designing demonstrations under Part D,4he programs shouW be consoli- 
' dated with at least '20% allocated for research ^ " f . * * ^ 

' . /' . * - ^ • 

• Shifts in policies and goals have bedn du^to fr'equent changes^h-administralors and reorgani- 
'Zations of USOE. programs should remain administratively in USOE's^Bureeu of Occupational 
and Adult Education 

• National arid multi-state problems have not beerugiven adequate attention under ^present 
(ii^tribution formulas, 50% of the vocational education fl&D funds should be reserved as the 
Commissioner's shaVe for attempts to solve these problems 

^ • Career education has been heavily supported with vocational ediTcation R&D funds, separate 
funding for career education R&D should H3e appropnated'and both R&D programs s?ioC»|d , 
be articulated 



Program Planning and Administration ^ - 

• 



R&D priorities have not been on a long-term, scientifically based schedule, systematic, open, 
^ cumulative, anjJ data-based processes for identifying priorities should be initiated at natfonal 
-and state levels 



Grant^nd corttract announcement and funding procedures are restrictive, a broad n>ix of 
announcement and fundirfg procedures"'should be used- at national and state levels\vhich pro 
vide more tirfie and multi-level stages of proposing 

• Women and mindrities have not been inv^olved in R&D enougVi, these groups should be en 
. couraged and facilitated to participate ^ . . 

• USOE does not have an efficient system for collecting and recording information on the R&D 
program, a national management information sy^tem.for vocational education R&D should be 
established ' ^ 

Institutions . ' ^ • - * 

• National B&D centers have served useful and essential functtgns, but have been less effective 
because of fundin^shifts, program purchase policies, and*being forced to compete with their 
constitufency, at least one" adequately^ funded national R&D center should engage inx:ompre 
hensive work on national and multi state J^roble'ms, policy development, high risk research, 
and disseminationln collaboration with others 
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• Curriculum coordination on a regional basis hSst)^en underfunded, fragmented, ineffective 
and duplicative, there is a need to coordinate curriculum, but if tt can't be adeqi/ately ^ 
f[nan<;6d at the regional level it should be done at a national center 

• RCU's vary in organization, functions, and effectiveness depending on sizfe, administratida 
and assigned responsibilities, RCU's should ^e funded at a minimum level in ever/ state • 
supplemented by size formula and federal, and state program matching funds, given broader, 
responsibilities for Parts C, D, and I and general' R&D management , 

^ Dissemination^and Utilization ' ' ^ - * * ' * 

Research reports have been accessible through ERIC and AIM/ARM but tKese programs have 
not been adequately supported, a comprehensive program should be ensured, put in one 
. . place, and^sponsored by. USOE/5OAE until NIE'S'ERIC can support it adequately *^ . 

• Many vocationaLedutators^are unaware of or un^blejo u§e R&D results and products, 
irrformatioil analysis program should be established to transform R&D^outcomes itttd* View, 

» targeted fol^s - * . . . ' 

• Adequa.te. dissemination and utilization programs have not been*given priority, dissemination 
and utilization should receive a significant proportion^of federal R&D^furvls, others should be 
funded for dissemination and utilization 

Critique of COVE RD ^ ^ . ^ 

COVER D chose the most rigorous criteria for assessment of vocational ediication R&D-impact 
on students^ Many feel that the intent of th^ authorizing legislation was not imfDact on students, but 
totjjild institutional capacity, train n^earchers, develop curriculum for emerging oecup^tions; and 
demonstrate new programs. ^COVERD's insistence that vocational education. R&D'should have left 
a measurable impact on students»cannot be refuted, hovyever, the term "measurable" may be the 
Anchillestieel'of the COVERD repert. COVERD did not measure student/achi^ement. It did not 
collect new data about vocational education, but depended mostly on secondl|ry sources. It did not 
conduct a systematic analyses of R&D reports developed under federal and state funding. Only 120 ' 
publications were cited in the COVERD bibliograptiy and many of the^ were reviews, journal articles, 
or COVERD'srown commissioned papers/ . / ^ 



Iffi 



It is stilk not known whether there was ijnpact upon students resulting from the 5,00Q prqjects. 
ere is^ COyERf) djd not find it. 



COVER D provided a focal point and forum for the discussion of wherff vocational §ducation 
R&O should ]0 next and the COVERD/Report makes some astute recommendations. Injthe final 
analysis, thesu recommendations represented the synthesized judgment of an august body of national 
Readers whot:onsidered the complex issues involved. Their recommendations.were taken into account 
in the newesdvocational education legislation as will be seen in a later section. 



• --^ - ■ . ' 

- ' Rand Report' ''^ 

Funding for the Rand Study qame'from the U.S. Office of Education (U^OE), an agency whose 
basic purpose is to-^. . introduce and spread innovative practices tn public schools."^ {Ran.d, 1975, 
{p. iii). Four programs were reviewed by Rand to identify how effectively those programs spread 
"innovative practices." The program includecl the. Element^y ^nd Secondary Education Acl 
(Title '1 1 1, Ihnovative Projects), Elementary and Secondary Education Act (Title V4I, Bilingual ProjectsV, 
* Vocational Education Act, 1968 Amendments (Part D Exemplary Project) \ and the RIght-to-Read 
Program. ^ ^ ' ' 



The Rancj/studiesbegan in 1973, F,ive volumes of their report were issued which describe the 
first year's effckts. Appendix D of Volum€ II T reports on career education projects that were funded 
u^nder the Part D of Vocattonal Education" Amendments of l'968. Topics discussed include, authoriza- 
tion of Part D project funding and^U.S. Offline of Educatioa planning and managehfjent strategies, role 
^of state educatibn^agencies in P^rt D career education prdj^ts, case studies^ and s^ntlijssls of case 
studies. '. V* * *M ' . -# 

^ Methods * ^ ' * 

CRand conductejion sfte <ntervievys nine project sites ahdT^ported on $ever}iirojects, three 
v\fere federally adrf»iniitered, two were state administered, and tw^vyere bot^iJederaL^nd statct-ad . 
mmistered activities. These rev^iewS'Were reported as case studies.^ • * . ' 



'Findings ^ . . * 

Findirigs of th^e Rand.Study 'on "first-round" (Jareer education programs ^ave been extracted 
from Rand's synthesis of case studies. *The Authors indicate they visited nine projects but only provide 
• .seven case studies (p. II l-l). \ « 

J Thepurposeof career education is stated as. "Children wefe to learn the connection betwe^ 
what they werfe beln_gLtau5!}t and wh*at they would have to know to earn ^ living." (p. \\ f I). ^T% tinc^ 
ing5 are quoted f^^bm the repprt in the follpwing section^ 

Initiation 

• Projects with more explicit and longer planning^eriods did better, second round projects 
are seeking to derive lessons from their predecessjprs. (p. 111-2 and 1 1 1-3) 

-M-;.. Jj . ■ 

• Our sample of fieldwork sites was so small thamve wer<fe unable to draw strong conclusions 
about the* preferred site of administration. (P. m-3) , *' . . . . 



^^ederal Programs Supporting Educational Change, Volume III, Th^ Process of Change (Appendix 
D. Innovations" in Career Education). Anthony Pascal et al., Santa Monica, CA. Rand, April 1975. 

^The programs were iptended to change the status quo, to res^ult in new^educattonal practices 
that have arf impact on .educational systems and students. % ^ 

^^Particularly the caf-eer^Mucation projects funded in 1970-72: ^ 
^ ^ ' '12 



A, 



. . the STiall sampig and various complicating factors rrlak^it impossible to render a juol^ment 
as to the relative efficacv4 of federal review state administration in the firsjt-rourtd^ar-e^edu- 
cation projects. '(p. 1 11-3) ' ^ • ' ^ - - ' , ^ 

• . . .\A^ell-fihanced middlesized (scJioqI) systems simply tend to go in fc^r i-nnpvation. They are 

^ constantly changjng.theTns^ves, at lea^t in non threatening ways, and h^ve ieafned to do it. ^ 

(p. Illv4l " • ; . » • . , 

• ^ ^ , * ^'^ ♦ • 

^ ' . \ . * * . 

• ... all T)f |he good projects also had 'good project direcftors. 'tp. J 1 14) • , , 

• . , . whenever possible USf^E^shouid work with the-LEAltseif.' (p. IU-bI • 
It is'highly desirable to ^nvolve sc^fopl'building personnel in the earliest possible phase of t>ie 

. nr^^-ont (p, III.5) '-^ • . ^ • , ^ 



prelect 

'Prepl?nn/ng hag-a large payoff, (p. 111*5) 



• Criteri^ to select schools for Rarticip.ation m the project varied so ex^tensivelV tha*t it is dif- * 
ficult to ,draw generalizations, (p.. 1 1 1-6) ' ^ , ^ ^ • 



ImplemWitatlon • ' * , • / 

• fn the better projects, materials tended to be developed on-site rTther than acquired else^^re. , 



(p. 11 



Staff development activities in some places consisted of two weeks in-seVvice straining and in' 
^ other places as little 35 three ho^rs." (p. 111-7) ^ ^' * " » ' " " , 



MSinteniUte and Adaptation \y 



Ma.rhtenahCe of the projects funded under VEA, Part in both .the federal- and st^ite- 
, administered versions was a significant problem. In many projects, especially tho'se adminis- 
. tered by USOE, start-up was delayed, (p, 111-9)^ 



there.Was little visible impact on junior and senior high schools: (p. 111-10) 



• It is rarely possible to identify, what actions the principals took that actually helped a project. 
Continuatiorf and Dissemination * ' ^ - 

• State support seems necessary for both continuation and effective ^jss^^ation. (p. 1) 



Conclusions 



The conclusions drawn by Rand fromtts case studies are quoted in this section. They appear on 
j^ges 111-12 and 13 of the RarKi Report.' ^ ^ ^ 

• jpareer education in its current develppmeht phas^is a fairiy weak and.mild program treatment 
/m^de up of standard components, ^t has qen&^wdl little opposition except by teachers of 
academic courses who have resisted ii^conj»fStmg'^''vocationarl'' gDnoerns. 
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iClany LEA'sxise career education to try to dthie/e other, and to^th^m more important, ends- 
fox example, humanizing the schools/mobiltzing thacortimumty, opening .educajion t6 inno ' 
vation, finding [o^g for the deserving, teaching life management, acquiring outside funds," and 
ifbproving vocational stj^fll training. ' • ' • \ 

Project^ that had a p^iojd of advance planning prior to thd cecerpt of the federal grant had / 
signifijcanlly highler levels of success. f * " J ^ ' 

A major career education objective-change iq f^ie^pprosoh-and behavior of teachers and 
•^counselors*-occurred oAl>5' in a few places 3nd there rather intermittently. 

. /■ . . . ' ' . \ ^ 

_ Lqc'aJ autonomy in the structuring of individual prgjects did not see'm to contribute much 
-to the Overall quality of the'profect design. • - * ' ' * ^ ' 

EaVly and^authentic partiC:ipatix)n by school level staff in^prqject conceptualization, however,; 
is necessary to engage the, cooperation ^of the actuarirhplementer^, , * ^ " 

TRe filter^^up strategy in which the receptivity to career-oriented education is to increase as 
studeats exposed in their primary years move up the grade ^adder is^tilKonl^y a hypothesis, r 
*. , ' * * ' * ' '\ ' ^ ' , 

Thadisinclination to confront seriously tbe/esistancp to career education in se^iondary schools 
may thepefore have Been a grave strate'gic-error. ' . i 

Use of outside agencies such aS*univers*ities and local action groups as intermediaries between 
USOE and the local scho6ls has not generaWy beefi a successful approach. 

, LBAs in comfortable (and main-ly subtRbah) situation's had more positive innovation experi 
ence. - ' ^ ^ , . * 

Projects in which^neither federal off iciaJs nor local authorities made suth attempt to monitor 
operations (erg., th^^odel CitieS-set aside projects) , did significaritly worse. * ♦ • 

For purposes of testing the effkacy^of ap jnfiQv.ative approach, proj'ects prqbably ought to be 
installed in the most promising and not tfee lYipst deserving schools, * ^ ' 

A tendency to develop materials locally characterized the best projects, bi^t thi$ m'ay well 
(^fade in import^ce as the^nation^ins experj^nce with careefeducatipn. 

Much'more serious and mudh m^^sustained attention? must go into curriculum revision if 
career education is to take in seconda/y schools. . * " ^* • 

SEAs will have to overcome their ignorance of and, skepticism ab^utrcareer education and 
shoultj facilitate career-oriei^ted innov5.tion in tocfal systenps by working for appropriate 
changesin procedures and •working at Interdistrict information exchange. • • 

True commitment by principals is^very difficult to attarn but seems to be associated with good 
school performance in this class of innovation. . *^ - . * , 

,For all practical purposes, state financial support is critical to the continuation of projects 
apd perhaps ou^ht to be pledged in advance of the federal commitment to a LEA. - . 



Critique . ' ' ' ^ 

* Jhe Randcase^studies review seven career edgcation projects^ their staff /linkage toother 
agencies, and^prpje'ct activities ifi narrative form, Wo "hard'^data is presented. Interview instru- 
ments and jnethods of collecting this type of information are no.t.fncluded in the ca^'e study reports. 



. 'Each case study \^ritt^n bsi^rjg a'f ictitious.narne, 6r, Allen B. Mpore, co-author of this - ^ \ 
report, visited 17 career education' programs in 13 states during'197l-72, thf-ee of which are - 
reported by Rand. These case studies and M(^ore/s experience ^re parallel especialfy the observations 
that most career education activities took place in the elementary grades, T$wer in^the middle grades 

* and even fewer at the senior fiigh levels, ' * ; / 

Using case studies is one meabodK)f analyzing educational programs/ However, this method puts • 
the burden on the authoj to synthesize and commuhicate what was observed' with accuracy. The ^ 
oniissipn of recommendatitjns and numerical data (e.g., structured interviews and questionnaires, 
h achievement data and budget infprmitlon) limits the usefulness at the report foe decision making, 

, Development Associates Report' ' 

^ , , The Develqp'ment Asisociates stufly was funded by USOE's Office of Planning, Budgeting, and 

• Evaluation tO' . . evaliiate the effectiveness^of thefirst^f^nd of federally administered Part D 
projects and theiV ccfmponents ip ter/nS of Tne extent to which student outcomes attributable to 

^ project ac\ivit;^es conformed to the legislative intent of Part D fund^" (p.,2).*Legislative intent, level 
of fiscal inputs,f'pro]eft objectives, emphasis of^i-oject activities, impact on students, continuation 
of project after initial fuji^ing, and some overall indicators 5f project accomplishments was to be 
reported. The legislativj^ intent for the projects reviewed was to focus on care^ education within 
' ^ - the boundaries .of P^rt D fundijig. * • . ' J • 

r ■ ' ^ , > ' - , \ • / 

G ' ' Methods . - v . ^ . ' t . 

Development Associa*tes began its work in June 1973, condqcted field^ork in February-May^ 
1974, and submitted life final report in IVlarch 1975. Of 61 ongoing pi-oj^cts, 50 were selected for 
on site review^ Reports and available do.cu^ft^Kfe were reviewed, interviews and que^5t4ftpnarres 
we^re used to gather data from SEA staff, LEA project f^erfionnel, and students. Data rwas gathered 
^ from approxinjfftely fivfe percent of the students and faculty in the 50 projects including; ^ 

• 4632 participating students tested • . ^ . , ^ 



• 4043 non^pjrtfcppatfng students also tested/' 
•\1433 teachers surveyed *^ , . . 



229^ counselprs surveyed 



Y 

ERIC 



Ma? Evaluation of Vocational Education- Exemplary Projects, Pan D Vocational Education Act • 
Amendments of J 968. Washington, D.C.: Development Associates, Inc., March 1975. 
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, ' FinWngs ^ 



Findings citfUfcom the Development Associates report tnclude: 

y A rfeviewof the ^ear-by-year input data revealed that in most projects in no single year did * 
. / - the Jnputs occur at the level pl^nned^. , it was found that in 37 out of the 50 projects 
/ visi^d (74%) the dollars expended we^rbelow theplannedlevel. (p. 4) 



s 

>^ • A review of the stated objective of the ^O«projects reveals that in man'y cases the activities 
caJled'for . . .;were not addressed, (p. 4) I * • * ' . 



■ERJC 



• Op a praject-by-project basis the impact of ^he progrem on students was small, with the bHik 
of favorable outcomes in each studeht group coqfined 'to a small groufj^oi projects, (p, 4) . ' 

^ \ 1* ' , * / 

, • With respect to elementary, junior hlgiyarid senior high familiarization activities, positive , * • 
i^mdications^of profect effects on stude^nts /occurred wheretrelatively more project funds were ^ 
expended on relatively fewer stutferrts>{p. 5) • / . 

• Weil over half of the teachers and counsefors surveyed in each of the 50 states visit-ed indicated 
in their judgment it was important to include career education in the school curriculum, (p. 6) 

• In general, neith6r"the' federally sponsored activities nor the fedei-ally expected student levef 
outcomes of the program occurred at the level planned. • (p. B) • ' 

i > ^ vv* 

• The definition of'key terms and concepts was neither precise nor .consistent at either the 
. federalor local levels, (p, 6) ^ • * " 

• Budgets^^nd^xpehditu re records typicajly were bas^pd on "lipe item" rather^han programmatic 
activity categories. ^Determination of activity costs was very tiiffi<^ult. *The difficulty was 
primarily a result of .the grant application and award»process which did jiot specify'^costs by , 
activity; only in the aggregate, (p. ,6)^ ^ - 

• The evidence stronglV^nd^tes^t^^ exemplary programs require considerable start up 

^ activity andttfne> Failure to anticipate this adequately appears tp have- resulted in the inability 
of projec!rto mfeet expectations, (p, 6) - ' ^ 

• ^Generally, pa'^ticipants i^n the projects were more exfiosed to visitors 141 their classrooms who' 
discussed careers, and went on field trips to learn about jobs, than nonparticipantst The data 
suggest that this quantitative difference ,in the number of such experiences was not sufficient^ 
to^produce a measurable impact on-students. Ratfter, it appears that such activitres need to 
be integrated 'into a wetl-plaWned and comprehensive effort, (p. 6) . 

• Continuation of activities beyond the project (the USOE, Policy Paper required that grantees 
make provisions fpt^continuing project activities after Part D funding was terminated) was 
disappointing: /" .^J 

^ * 14 projects would termiifi^e— /70 gontinuation funding 

19 projects woulc/c6ntinLue5o/7?e activrties , 

6 projects would continue ac|ivitie3 

' 9 pro'iects pvoufd expand and continue actmt\es ' - * 
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• "Only 26%of.th6 projects reported hawng students, in all levelsof project activities- larqest 
enmllments were in the elementary and middle grades; smaller'enrollmentlVthe high school 
grades {10-12;: , • 

' \ ' . , * . ^ • 

46% of the elementary enrollment averaging 2,000 per project 
. / 47% of the middle grade enrollment-averaging 1,400 per project ' . " • 

, 35%ofhi^^chooi enrollments or eSO'per project ^ ^ 

Nevertheless, 44% of the funding was used at the high school level,- as compared to 26% for * 
ejeipentary and 29% for middle Schools. • . = ' 

Conclusions and Re(^mmendation^ * ' ' 

. Development Associates^concluded that most teachers felt that career education should be an 
important part of the curriculum, therefore the first year Part D program had a substantive effect 
bvenso, they concluded that activitiefe' and. outcomes did not reach expectations. Clearly defined ' ' 
objectives, definitions, and managerial requirements are needed at project and fSderal levaW: More 
^pecitically, Dev^opment Associates concluded (pages 6-7) that: 

^ Thedefinitidn of key terms and concepts was neitherprecise nor consistent at either the ^ 
. federal or local levels. . * - » ' * ^ 



• Budget?.^ expenditure record* typically were based on "line-item" rather than'prodram^ 

matic activity categories. Determination of activity costs was very difficult. ' ^ 

____ » ' ^ ^ ^ 

• Similarly, USOE did not use fiscal data as management indicators,- , 

• The eWdence strongly indicates that.exemplary>rograms require.considerable start-up 
activity and time. . ' " ' , ' 



• Generally, participant m tbe projects we/e exposed more to visitors in their classrooms who ' . 
discussed careers, an^ went on more field trips to learn about jobs, than non-participants. ^ - 

-. - "■ > • . ' • • . ■ "> :. 

• The number of different approaches to building a brjdge between schooh and earning a living • 
undertaken^by the first round projects was limited. Fewer than half of the proleqts had vvorlc*^ 
experience or skill training activities. # ' - 

• / . . * • • ' . 

• The prinriary foQus of round one was elementary and secondary familiarization and orientation 
.IVIost of ttie total funds and ttiost of the student participa'rits were engaj^d in such jrctivjtie^. 

• Jo assist college ^d non college bound ^tudents in obtaining emplbyment, the USOE policy 
paper indicated that projects should provide specific training in job entry skills to students 
not previously enrolled in vocational programs just prior to .the tiq^e that they feave school. • 
The data appears to indicate that most prpjects extended such trainirfg only to the non-coileqe 
bountl. \- 



The student responses indicate that guidancq and counseling at the'l\igh schQpl level was vjewed.- 
by stotfents as helpful to them. The data also indicates that a great many students did not feel 
they had suffici-ent opportunity to receive assistance from their counselors and they would hav&.' 
liked to h^ve increased their contacts. ' - ' " 



• A final point relates-to the vyhole area of the management of'educational projee^. Wtfile tj>is- 
^ was not a management practice study, many»of the findings and conclu^ons appear *to relate 

* directly to martagement issues. The points raised above pertainmg to a^jvanced planning, the » 
* * clarity of program and project objectives,, and the use of fiscal clatd are nianagement questions 

I which relate to attainment of student levef outconies. " * ' ^ 

Critkjue . ' , • - . ^ . * 

• ' . " , ♦ ' ' 

The Development A^ssociates report is the most systematic and data bas^d of the five reports. 
It was systemati9 in sampling, data gathering, and reporting. Large^amounts of data were collected 
^ from nop participarits, students, teach^rs^nd counselors involved^ carBor education projects through 

out the nation during 1973 74. ThisJnfo^nation was collfected at the conclusion ©f-^irst-round" 
^ funding f-or Part D (VEA 1963 as amended in 1968Kexefnplary projects in career education. The 
report did not include reconpHnendationS, a serious limitation of the report. . . 

' , GAO Report^ • ^ 

The^Comptroller General Report, often referred to as the GAO report, evaluates vocationaf edu- 
' cation programs authorized under the 1963 Act (P.L. 88-210) and the 1968 Amendments {P/L 90- 
576). The Comptrqfler General ". . , focuses on selected aspects of secondary and post-secondary 
vocational education in seven states visited during the 1973 74 school year and discusses some under- 

• lying factors tha^inhibit attainment of objectives" (p. 1)v ' ' 

Methods ^ ^ . ,^ 

The.GAO study was conducted in seven sktes. California, Kentucky, Minnesota, Ohio,- 
Pennsylvania, Texas, and Washingtc^. Data gathering procedure^ included interviewing federal, ' 
req^ional and state vocational educayon personnel, reviewing reports and literature, and making visits 
to states. GAO examined legislating^ directives, regulations, plans, repG^rts, and other docu- 

ments- -/ '^. '" * ^ 



Findings' \ ^ ^ \ , ^ ^ 

GAO reported findings which focusL upon the role of federal funds, planning fo? use of 'funds, 
distribution of funds^use Qf training resources, and relating training to employment. • 

^ Role of federal funds. The legistatioQ intended that f^eral funds v^oult) be distribufed to local 
agerK:jes to increase funfjing, give priority to special need categories, provide programs for new job . 
.^|:^portunities, and -increase participation. GAO found that large amounts of iederat funds were us^d 
at the sjtate level for administration, the ratio of local and stale matching declined,*special groups weri6 
not given a high priority, and enrollm^ents had not increased proportionate to fgnljing. ' 



^ Report to tfie Congress: Wtiat i^ ttie f^le of Federal Assistance for Vocational Education? 
•Washington, D.C.: Comptroller General qf the United States,,December 31, 1974. ♦ 



Planning for use of funds. Better planning would result in better programs. GAO found more < 

^nZir'^H P '""'"^""'^ "^^ds assessment, fragmented organization, problems with advisory 
^ councirs, and insufficient data. ^ , • *. auvi;>«iy 

Distribution of funds. Federal funds have not always been targeted to needs or initiatives-caiied 
. for m the )^^x^on. GAO found funds being distributed generalfy rather than to high need, without 
regard to needs data, and without regard to. local edJtation agency ability to provide its own resources. 

Use of training resources. The rano^of training resources available to state and local agencies ' 
have no,t been fully utiUzed. GAO fourf5 that schools want to control the facilities they use re- 
sources haven t been hventoried, costs haven't been compared, schedules haven't been provided, new 
/construction is prefer^d, and sources of equipment and suppHes haven't been explored 

ry^.J-^'^^'^J ^'■a/n/W^ ?o>.«e^en?. Federally-supported accupational programs haven't addressed 
.changing occupational requiref^TeTits. Students in traditional programs can't always find jobs GAO 

^^V'?°' ' 2'*' '^^i"9 used, work experience often not used, and guidance, place- 

ment, and follow-up not being used adequately. • ' ^■^'MOLC 

\ • .*^ther barrier5,Such as age, sex, and entrance requirements, were found to restrict-'^'ccess to 
training and emtjjoyment. * 



.Conclusions and Recommendations ' 

po n^c^ '■^P?? recommends specific action by DHEW that also could influence vocational education 
K&u. bome of the recommendations are: 



♦3- 

2, 



Expand management evaluations to state and local vocational education-programs supported 
by-federal funds (p. 1t>2^. » I hh 

Expand working partnerships among all institutions providing occupational training at all . 
levels (p. 102); /* , 

3. Expand efforts to develop labdr market data in a form which will better enable vocational 
, . planners pt state and local levels to m^tch occupational training with manpower needs by 
working cooperatively^ witl/the Dep.artment of Labor, and provide technical assistance to 
states for the training of vocational planners in the use of such data (p. 107) 

GAO recomrnenc^^ that fewer federal funds to be(jsed for state-administration of vocational edu- 
cation and that federal funds use^l at the state level principally be for planning. It recommends betfer 
matching or large^s%sides fot special needs-programs, mor^ cooperation with othe» agencies on pro- 
grarns andtacilities and better matching of programs to the job market. It recommends requiring ' 
work experience, placen^ent, and follow-up as pect m programs. 



<1 



n summary, jhe GAt) report points out the need for closer monitoring of federal fund^ it all 
levels and encoar^ges cooperation between and among agencies to provide vocational education for 
youth and adults. Even thpugh the report is based upon study of only seven states and is directed at 
Part B programs,'-fecorT]mendation5 have applications to vocational education R&D because of its 
close connection to operating programs. ... ^ 
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Critique 



GAO cautions that the report is based on data frorn'only seven states an(j may not be applicable 
to other states. Nevertheless, other states have had to defend their prograrws on the basis of the 
limited findings of the GAO Report, The broad generalizations and sweeping statements by GAO 
have not been thoroughly documented in the report, therefore many state leaders have been inclined* 
to refute the conclusions. At issue is whether federal funds for vocational education are to be 
regarded as program support or for catalytic purposes. « • . - -c 

/ ' ^ 

DOL/MR&D^eport^ , * . 

The Department of Labor requested the National Academy of SciedceJ^-National Research , 
Council to review, assess and makel-ecommendations about the manpower researctiand development ' 
program whLeiTTTad^been in operation since the passage of the Maripower Deveitop/nent^d Training 
Act (MDT-^) of 1962, The DOL/MR&D Committee took an "impact On policy" approach as it 
examined manpower research and development, ' ^ ^ 



Methods 

The committee (D&L/MR&D) used^^everal methods and sources to gather data on manpov^r^r^ . 
research and development. Five subcommittees were formed to synthesize inforjnalion for the report. 
The subcommittees studied such problems as tp. J 41 ) : . ^ ~ ' 

• generation and initiation of R&D projects; ^ ^ ^ ^ 

• manpower R&D findings; 

• Utilization and dissemination of R&D findings and results; * y 

« 

• R&D organization aad management^ , . — » ^ 

• future manpower problems and policies, - ^ . " 

Twenty papers v/ere commissioned to summarize information on selected toj^ic's. One of these 
topics was developed by Dr. F, Ray !\^rshall, current U,S. Secretary of Labor. 



The DOL/MR&D staff prepared memos and refpOrts for review and discussi/)n by the larger com 
mittee.^ Interviews were conducted with 26 Office of Manpower Research and DeN/felopment (OMRD) 
staff members, 138 current and former Departmentof Labor officials, 49 personsjepresenting state ^ 
or locSI manpower programs^ 18 individuals representing private SrOups, 95 personsj^ho were involved 
in OMRD supported projects, ,17 university based researchers, and 32 other researchers responding to a ^ 
survey letter. Archival searches of the OMRD files were conducted for information and data On 
MR&D. Special meetings, observations and conferences were attended by DOL/MR&D committee 
members to obtain information and data. The committee report lists approximately 300 documents 
on research and related materials that were reviewed by the committee/ 



^ Knowledge and Policy in'Manpower. A Study of Manpower Research and Development 
Program in the Department of Labor. Washington, D.C.. National Ac^Semy of Science, XQ75. 
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In Its st^dy of ten ye^rs of DefDartment of Labor manpower research and development, funded 
at about S250 million, the DOL/MR&D committee reported that there w^re four arSas where sub- 
stantive acconiplishments had'been made {pp^^22;^): V , ^ 

1 The collection arrd an^^ysis of labor force data pefmits^more thorough and detailed examina- 
tion of the econorpic:'ir\stitutional, psychological; apd social faptors underlying employment 
, • success than ever before possible.- 

2. . The development of new theories to illuminate complexities of labor market operations, 

especially theories that help explain relationships between market imperfections and indV 
^ viduals' employmentexperiences, provides new grounds for discussing the role and effective- 
ness of manpower policies. ' • 

3. 'Program models and techniques for serving the manpower needs of the disadvantaged have 

beeft designed and implemented.' ^ ^ 

4. rvtew methods for assessing manpower policy and program effectiveness have been refined 
and applied, 

Whilejhe Committee noted the success with labor force data,-r)ew theories on labor market 
^ operations; program modules and techniques, and methods of assessing manpower policies, they 
suggested that there was need for manpower R&D to: 

^ Review, analyze and synthesize diverse and scattered manpower studies for implications 
and contribution to theories and policy; ' 

- -2. Examioe cultural factors of minority and majority groups; 

^ ' Sofve the methodological problems associatecl;with job search behavior; 

^ 4. Identify, isolate, and eliminate racial/job discrimination variables; 

- . .\ Strengthen thev^process for studying nontreatment variables that may have influences on 
. ' ^ manpower research; and • i ' - 

' * * * 

6, Reopen the study of technological^ chafnges^on people and the lab^^market. 

The Committee also noted that the Department of Labor apparently regarded its R&D activities 
^as a service function, managing it as an adjunct to operations apd coftseqaently failing to focus on 
Wng-r^nge policy development^ in the broadest sense, {p. 33) 

♦ 

4 

Conclusions and Recomqiendations 

* * * /- 

The Committee made several recommendations related to program content, scientific capabilities, 
program management, utilization, and R&D resources and budgeting. These recommendations are • 
listed here by category {pages 35-47): 




Program content There are opportunities to build upon current knowildge, to develop new- ' 
knovvledg^related to confirtuing issues and on developing issues, and for increased use of social 
science methods. ^ • * A > 

. ^ ♦ f , 

Scientific capabilities. The Committee recommended snoall grants for doctoral and post doctoral 
work, evaluation of grantee performance and expansion of conti'nuing grants, and \oQsideratton of a 
national center. • ' ' ' \ 

, Program managenient, .The Committee recommended long term planning inv/lving others, irn- , 
proved mechanfsms for performer selection, anci improved OMRD staff capability through, recruit- 
ment and trairrfng/ * " * 

■ - -i "'. . • • / ^, ' ^ ' ' 

Utilization, The Ccypmittee r^ortimended a number of techniques to improve the capability 
oiONlRP and DOL to achieve utilization of R&D based solutiohs {?y CETA prime sponsors and 
others, /ncluded in the i^commended techniques are studies,\surveys, demonstration, technical 
assistance, trairiing' papers, and workshops. 



R&D resourjces ' ' - ^ - 

range resource 

^- 

'^ 

Critique • ^ 
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es and budgeting. The C^rr'rfmttee recommended an extensive analysir^fjong 
uiremenPts to be repeatec^very five year\i|, * , * 



' The DOL/MR&D study was a systematic review, analysis/and synthes^pf R&D impact on 
Department of Labor policy. If appears that by dividing iato subcommittees they were able to 
act<jaNy analyze data from ^pproxi^tely 300 project reports. As a result, they were able to describe 
accomplishments of the manpowetj4^&b program, identify unmet R&D needs, and mak« recommenda- 
tions to strengthen the R&D program, 

• — ' - . 

J r r SYNTHESIS OrREPORTS '/^ 

Unvsnnding the-common threads from these reports has been a delicate process. 5^raning 
the|e threads into a useful fabric >ias beer> mpre artful than scientifid The fabric of this synthesis 
is \^ven from the assessments of R&5 focus,.accQmpf^$hrrients, quality, impagt and n^anagement. 



^ Pocus of the R&D Program 
\ ' 

The COVERD Report i? critical of vocaticnal education R&D for failing to fpcus onnhe larger' , 
philosophical policy issues confronting vocational education dunng^he ten-year period. On^he v 
other hand, tf^ Part D program reviews (Rand and DevelofJment Associates) were of exemplary 
programs that were ende^vqring to demonstrate the efficacy of career education, a majer new concept. 
Wrilat COVERD was concerned*ab6,ut'was the alleged failure of vocational educaftion R&D to contrib- 
ute to the clarification of the major purpose of i^rocational education. COVERD called for , . defm; 
mg objectives, measuring the actual benefits of existing programs,'and initiating exploration of new 
subjects in vocational education, rather. than simply reacting to problems of existing programs . , /' 
(p. 3). A similar observatioQ is made fn the DO'L/MR&D Report which.pointed out'that manpower 
Ff&P tended to be a service function adjunct to operations, /ather than/^oncerniing itself with poHcy. 
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R&D Accomplishcnents 

' . ' ' ' ^ 

While lamenting the lack of impaction students, COVERD singled out nine projects that had 
made significant contributions. They also reviewed the literature in nine major areas. COVERD 
criticized USOE because reports of prbjects conducted under the legislation couW not be tracked,, 
a criticism not warranted according to otl^er'sources. .The nine major areas chosen for review by ' 
COVERD (i.e., career development and guidance, studehtwith special needs, characteristics of siiJ- 
dents, teacher education, instructioi|al techniques, curriculum development, labor market supply 
and demand, administration and evaluation) were of major concern during the ten-year period under 
review. COVERD found other benefits (e.g., accumulation of knowledge, R&D capacity) of vpca-j^ 
tional education 'R&D, but stated that even these haye not been m^easured, documented or validated. 

Rand and Development Associates founfl that career education exemplary projects, while making 
contributions to students and st^ff while in progress, did not contin\je as reqjuyed in project agree- 
ments They (^id find mdreSgothusiastic reception to these programs in elementary ancf middle school 
levels, more career guidance activiti^;and better acceptance of career education at a concept. The 
^ DOL/MI^D Report also alludes to%TnTxed'Bag of accomplishments, with the most accomphshment 
on program prqc^ss and less on policy Jntfprogram impact. GAO did not address R&D accomplish- 
ments. ' \ ' ^ " 



Quality of R&D ^ 
* - ^ - ' . • ^ 

Dai/MR&D comes clean on the quality issue whep it says that the extramural manpower R&D 
IS varied but generally gooS. ^COVERD is much less direct on the issue.^ It allows that there has been 
an accumulation of knowledge, development of new programs how in use, commercial publication 
of materials developed, arlcl an increase in vocational education Ip&D capability. It reviews the insti- 
tutions thaf have been built and describes theTif as underfunded but generally good.^ It cites nine 
projects, mentioned before, but these e^ianated from sqftie rather'direct, personal sources of the 
committee. The fact seems to -be that COVERD took testimony, studied reviews/visited sites, and 
speculated on fDossible impact^ but didn't actually study many research reports that had been ' 
funded by the programs under revievy. Rand land Development Associates did not address the quality 
issue, except in indirect way? (e.g., project piannfng, management, treatment effect*, continuation). 

Irfipact of R&D ♦ ^ . 

— ^ 

The CO\/ERD study group foundlittre evidence of impact, a finding that was corroborated in ' 
thefland and OevelopmentAssociat^:lty digs of Part D career education projec^rThese findings 
were .confounded by a couple of factors. First, the COVERD group admitted that ftjacke(^ the evalu- 
ative data or models and believed that there was some impact even though it couldKhfthejraeasured 
objectively • So, in addition to lamenting thQ lack of evidence, COVERD criticized|ielaclc'of impact 
^ measures. Development Associates found little evidence of continuingjmpactby exemplary projects 
' on participating school system^,or on other school systems. Interestingly, lack of impact on some 
aspects of the ijianpower program is the complaint of the DOL/MR&D^epor^ 

Impact measurement is difficult at be^t, but it is hard to dfeoy th&objective evidence in the 
Development Associates study wher^'actual^st data were gathered from participating and non- 
participating students. Developipept Asso(fiates suspected some cross contamination in treatment 
and Band judged the treatment* itself to.tre weak. 
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Management of R&D ^ jf^ 

All of the reports'dwelled upon hianagement. Many of the recommendations are directed to this 
aspect rather than the substance of vocational education R&D. It may be easier to comment upon 
the managem^Qnt than on the substance of the program. Nevertheless, COVERD blames the deficien-. 
cies of vocational education,R&D on "... a lack of coherent policy, administration, and leadership 
in vocational ^ucation R&D program" (p, 2). While the size of the staff, monitoring load, qualifi- . 
''t^tion^ of the^staff, management systems used, and pertinent regulations were implied to be the' . 
underlying problems, empirical evidence or adequate description^ of these are lacking, 

Rand suggested that exemplary programs needed better planmiig and stronger local leadership. 
Development Associates^found that line item budgets didn't permit knatchmg accomplishments 
to expenditures, and that underexpenditures were common, D'OL/WR&D found that the federal 
staff, declining in size and experience, was adversely affecting the Cra.L manpower R&D program. 

■ ■ 

' FUTURE EVALUATIONS OF VOCATiOWAL EDUCATION R8fD 

OT the five studies under review in this paper, the n:iost comprehensive treatment Qf vocational 
education R&D was by COVERD, Conclusions from COVERD were corroborated or paralleled by 
conclusions in the other reports. COVERD criticized vc/cational education R&D for its lack of impact 
on both policy issues and ultimate target groups. But/vvhat has been the impact of the evaluations 
themselves? Did Congress use COVERD's recommencfttlons in the new legislation? Did USOE and 
other agencies change their administrative procedures? Did the RCU's behave differently? Do others 
krtow about'COVERD's findings? Did the COVERD Report meet its own high standards? 

i Impact of the Evaluation Studies > 

As mentioned previously, a draft of the GOVERD Report was made available to Congress rather 
late in the committee hearings on new Vocational education legislation. On October 12, 1976, only 
a few months'after the issuance of the COVERD Report, the Education Amendroents of 1976. (P. L: 
94482) w^re enacted into law. It is,clear that Congress h^d some of the same concerns as COVERD 
because P,L. 94-482 will result in the following changes: ' - 

1 . Research, demanstr^tion, and curriculum development programs are to be consolidated 

2. Funds^are to be reserved for the Commissioner to use m solving problems of national 
\ significance ' 

1S\ ^ plan is required for establishing priorities and coordinating w(yk on national priorities 

4, A management information system for'funded projects is to be developed 

5, National center for research in vocatiorral education with six broad functfons is to be 
established 1% . % ' ^ 

6, Research coordinating units are authorized and are to be given broad responsibilities for 
vocational education R&D in the states 

* * , f 
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7. Disiemination is named as one function of the national center and as a function of the - 
stale research coordinating units. 'Dissemination is given impetus with the requirement " 
that contracts and grants be required to demonstrate probability of-results within five 
years {thus requiring atlention to dissemination at the outset of projects) * 

Other COVERD recommendations have been dealt with by USOE, NIE, and other government 
agencies^Some of this progress was being made even before COVERD ha.d made its recommendations. 

1. Research coordination council at the Assistant Secretary's level was.established in other 
legislation - ^ 

2: Systematic, open, and cumulative priority-setting activities have been developed by USOE 
ar^ ere operating within the constramts of Federal Procurement Regulations ^ 

* * 
3* Both contract and grant procedures have been used by USOE, effort has been made to ^ 
announce priorities well in advance oi grant announcements, and bibliographies hav^ 
'c been made available'to help Part C and 0 grant applicants 



The cQntract for the f RiC Clearinghouse on Career Education was awarded by NIE to 
The Center for Vocational Education at The Ohio Stpte University where the'USOE- 
sponsored AIM/ARM Project has been located, the two programs are being articulated 

Both AIM/ARM and the ERIC Clearirtghouse are undertaking information analysis 
activities 



6. Some curriculum development projects j^^e included dissemination plans, USOE has a 
National Diffusion Network to disseminate approved products and programs, and NIE has 
several dissemination programs underway in which vocational education can participate 

7. Vocational education R&D remains with USOE's Bureau of Occupational and Adult 
Education 

8. Women and minorities received considerable.attention in the new Amendments 

The general public hds had only a fleeting glimpse of the COVERD Report in press releases which 
highlighted the "lack of JnSpact" of vocational education R&D. 

The COVERD Report l;ias not had great impact on state-level vocational education R&D yet. 
individual RCU's were invcMvfifl in the study as subjects or in providing testimony. As a group, the 
RCU's included COVERD presentations and discussions in three of their annual meetings. But they 
have not had mechanisms to deal collectively with the issues raised by COV ERD. the impact of 
COVERD wiJI be felt by RCU's when the Rules and Regulations ^are impfementedJ 

. Improving the Impact of Evaluation Studies i 

An evaluation of vocational education R&D has its most im'por^nt impact upon policy (as in 
Congressional fegisl^ion), for decisions -{as in federal agency administration), and ior pjanning (as*in 
an agency or organization at any let^el). So the qu»t«?n is, how can the maximum impact be made? 
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An evaluation study of vocationat education R&D should address the "major policy issues, be in 
phase with legislative calendars, and be credible to policy makers. The next national evaluation of 
vocational education R&D should be planned and scheduled soon sd^ there \^\\ be time to complete 
the studies needed. It should be well designed with clear objectives. The criteria-by which vocational 
education R&D is to be jadged should be explicit enough to permit instrument design and data col- 
lectioJV^-J'rovisions should be made to package, and disseminate the evaluation results to appropriate 
audiences. The total evaluation ought to be programmatic, linking several projects foqether so that 
the summa^ive evaluation is broad based. ""^ 



Some Dimensions of the Next Evaluation 



Whether the ngxt federal vocational education legislation included adequate provisions for 
R&D may depend upon the conclusions and recommendations of COVERD M * " 

^ The next assessment of vobational education R&D should be'tlmety, dccorate/empirically-based 
and credib4^.To b/credible it must include input from those affected by and tjnvojyed in vocational ' 
education R&D. Such an assessment must be .comprehensively planned, include several projects 
articulated jnto a programmatic^ffort, and synthesized into a report worthy of implementation 

Actors in a comprehensively-planned assessment would necessarily/include a sponsoring agency, 
a steering committee, several projects, and an information system, Criteria for selecting actors in ' 
Ihe assessment p'rogr^m should include (1) independence, integrity, (3) capability, (4) knowle<ige 
of the field, (5) available resources, and (6) access to data. Some of the sponsoring Igencies and 
organizations which could serve in some of thesS roles includ'e the U.S. Office of Education, the 
National Institute of Education, the Geiperal Accounting Office, the National Advisiory Council for 
Vocational Education, and the new national center for research in vocational education. 

Comprehensive Planning - ^ ^ . ^ 

It would be extremely unfortunate if planning for the second assessment of vocational educa- 
tion R&D was delayed until 197g\ Planning, to be comprehensive, must get underway immediately. 
Otherwise, evidence of^mpact will still not be available and the final report will again be a retrospective 
survey of soft, secondary data. - ' • ^ ' 

A steeriQg congrmittee should be converted, possibly similar to the original Committee on Voca- 
tional Education Research and Development. This committee, with adequate support, would^plan a 
comprehensive, programmatic approach to the assessinent. Evaluative models should be examined 
and data needs determined. The steering committee couici then recommend a series of projects, to 
be conducted under separate contracts, which would be the data source for the final evaluation by 
the steering committee or its successor, possiblv^ named COVERD II. The sponsoring agency, with 
these re^commendations, would issue the necessary Requests for Proposals and monitor the projects. 



""COVERD II is the authors' designation for a second assessment of vocationaleducation R&D 
as recommended in the COVERD Report (p. 3). Such an assessment should be a continuous process, 
not . . convened every five years . . as recommendecfm the COVERD Report. 



26 



• 



Programmatic Effort 

' - \ 

The n&xt assessment should be the result of programmatic effort. VVhy programmati^c? Because 

no one study ta« adequately coH^t, process, and analyze all the. data required for such aJ^omplex and 
large scale assessment. Programmatts^effort has the goal of improvement-and ch^ge, not maintenance. 
It is aimed at reducing the discrepancy between expectations and observations artd provides a means 
.to-aljpcate scarce resources. For a more thorough examination of issues in progra^tmatic R&D see 
Pm|,n.er andltWalkerf* - ^ " / 



Some of the activities require longitudinal approaches. Others depend upon'^e results'o'f 
divergent studies. Articulation of these divergent studies must be planned in advance. Convergence * 
of results will be the main job'of COVERD 11. 

/ . ■ - 

Articulated Projects 

/ J. 

^ Assuming that the programmatic effort is done in parts, rather than a single massive -effort, 
the steering committee will need to specify the separate projects^needed. Some of these might include: 

Evaluation of the quality of a sample>K^D projects in terms of design, conduct, reports, and 
impact \ . ^ ' ' . 

• Assessment of the impact of specffic R&D projects upon intended target audiences 

• Development of techniques for assessing whether funded projectsTiave impacted upon students 
as required in P.L. 94-482 

• Development of other techniques to assess the impact of projects funded before the require- 
ments in the new act ' ' ^ * . 

• Syntheses of completed projects fanded under a specific priority to^determme what progress 
ha3 been made on the priority by tHP sum of the projects : " ^ 

• Longitudinal data collection to detect differences in the vocational education programs and 
, products traceable to R&D-..__ 

ir 

« 

• Arrinformation system capable of tracking priorities, contracts, and products, ^ 

These projects, a^s part of a programmatic effort, should be carefully phased and sequenced so that 
one contributes to the other at the appropriate time. 
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Synthesis of Findings 

The findings of the articulated projects ultimately need to be synthesized so that COVERD II 
can draw its conclusions and recommendations. Wheth^er it is practical for the steering committee 



*Pratzner, Frank C. and Walker,. Jerry P., Editors. Programmatic Besearcfi and Development 
in Education: Positions Problems. Propositions, Columbus, OH. The Center for Vocational Education, 
The Ohio State University, June 1972. . ^ ' ' ' ' 



to serve as COVERD II must be examined, but there should b(? some'common membership to ^ 
facilitate continuity of purpose and synthesis of activities.,- 

^ COVERD II win need competent staff support and consultant help; COVERD 11 may wish to 
commission experts to examine issues and problems in Hght of the findings of the various.stud'ies. * 
Taking testimony from representatives, conducting surveys, and making on-site observations will 
facilitate field inout.Jhe advantages that COVERD II will haye ^re plans, systemmatic observa- 
t'ons, empirical data, timing; and more knowledgeable involvement by ,the field. 



Dissemigation and Utilization - - ^ ^' • 

z-^- , ^ --^^ • • 

' With the widenrtvolvement of the field and a scfteduled pompletion, COVERD II will help 
^ Congress and the R&D comn^unity through the first tv^o stages of adoption ^awareness and interest. 
If the COVERD jl report is well done, it will stand up to the evaluation and trial stages of adoption. 

The dissemination and utilization Strategy for the evaluation report must be included in the 
initial plan. Such strategy must includ^activities designed to create awareness apd interest, packaging 
of information to facilitate user evaluation, trial, and adoption. For example, recommendations must 
'be supported by findings and conclusions that facilitate evaluation "of the recommendations. The 
, recohnmendations also musrbe stated so that appropriate target audiences can use them. 



/n Summary ^ ^ 

» 

The first COVERD Report and other evaluative studies got the attention of Congress and the 
educational community. Because the conclusions and recommendations were not coijiplet^ly accept- 
able to everyone involved or affected, the ne^t evaluation studfes will receive even more 'critical < 
attention. Therefore, it is extremely important that the evaluation effort be methodologically sourtd 
and on the right issues and problems. Early planning of articulated projects in a programmatic effort 
is essential. ^ 

The vocational education profession is general and the R&D community in particular wiW be 
interested in the planning for and progress of subsequent evaluations of vocational eddcation research 
Because of this interest, the agency or organization charged with'the responsibility of future evalua- 
tions must keep vocational educators informed. They can expect\o be continually monitored by 
vocational educators in the future. 
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